If you’re in the field of analytics or data science, you’re likely well aware that data is being produced continuously—and at an increasingly rapid pace. (You might even be tired of hearing this repeated!) While analysts are typically trained to work with structured, numeric data in table formats, a significant portion of today’s data boom involves unstructured, text-based information.

Unstructured data represents 80-90% of all new enterprise data, according to Gartner.

Furthermore, it’s growing three times faster than structured data.

Topic modeling is way to identify themes/semantic patterns in a corpus (complete document).

Topic modeling finds the relationships between words in the text, thereby identifying clusters of words that represent topics.

It is a like an amplified reading, a way to discover themes you may not see yourself.

Glossary:

Corpus: Group of documents

Documents: Newspaper, Blogpost, Tweets, Articles, Journals, Customer reviews etc.

Uncovering Hidden Themes:

Topic modeling helps discover latent themes and patterns within unstructured text data that might otherwise be missed, providing a deeper understanding of large datasets.

Efficient Information Retrieval and Organization:

It automatically organizes and groups documents by their main themes, making it easier to find relevant information and creating a manageable structure for large text collections.

Supporting Data-Driven Decisions:

By identifying prevalent topics in customer reviews, social media, or research, organizations can make more informed decisions to improve products, services, and strategies.

Automating Text Analysis:

It automates the time-consuming process of manually reading and categorizing large volumes of text, increasing efficiency and reducing human effort.

Enhancing Research and Discovery:

In academia, topic modeling helps analyze research publications to reveal trends and key topics, thereby streamlining the research process and potentially leading to new discoveries.

Improving Customer Experience:

Businesses can use topic modeling to analyze customer service emails or feedback to understand major challenges and concerns, allowing for targeted improvements to service delivery.

Latent Dirichlet Allocation (LDA) is one of the most common algorithms for topic modeling. It is guided by two principles, that:

• Every document is a mixture of topics
• Every topic is a mixture of words

Collect the text data

Examples: - Taylor Swift’s lyrics (there is a Taylor package in R)

• Spice Girls lyrics

• BBC News

• Gutenbergr package

• quanteda package

Before modeling, we preprocess the data to put in it in a tidy format by:

• Tokenization (splitting sentences into words)

• Removing punctuation, numbers

• Removing stop words (like the, and, is)

• Find document-word counts

A matrix that represents the frequency of each word (term) across all documents.

We can cast a one-token-per-row table into a DocumentTermMatrix with tidytext’s cast_sparse().

- Rows = documents;

- Columns = terms/words.

We can then use the LDA() function from the topicmodels package to create a topic model.

• Extract top keywords per topic.

• Label the topics manually (e.g., “Customer Service Issues” or “Product Features”).

• Visualize using tools like: ggplot2 package

Summarize the result Identify customer pain points Track emerging trends etc

We will make use of the following packages

• tidyverse

• tidytext

• topicmodels

• tm

We’ll be working with the BBC News dataset, a collection of 2225 news articles published between 2004 and 2005, covering five major categories: Business, Entertainment, Politics, Sport, and Technology.

The goal of this project is to combine all the articles into one dataset and apply unsupervised topic modeling to uncover the hidden, underlying themes within the news stories. By analyzing the text, we’ll reveal the most prominent topics that dominated the public narrative during that time — all without manually assigning any labels.

The BBC News dataset contains three key columns:

• Title – the headline of each news article

• Description – a brief summary or excerpt from the article

• Category – the labeled topic (e.g., Business, Sport, etc.)

For this project, we focus on the Description column as the primary source of text. This is where we perform all of our text cleaning and preprocessing — removing punctuation, converting to lowercase, tokenizing, eliminating stop words, and so on.

Once the topic model is built, we use the Category column as a benchmark to evaluate how well our model’s discovered topics align with the actual labeled categories. This gives us a way to check the quality and accuracy of our model.

We start by using the clean_names() function from the janitor package to standardize all column names (e.g., convert to lowercase, replace spaces with underscores).

Then we removed duplicate rows based on the description column using distinct(description, .keep_all = TRUE), so each news article description appears only once.

The data had 141 duplicate rows.

Let’s tokenized the description column using unnest_tokens() to break down each news article into individual words.

Then use count(word, sort = TRUE) to calculate the frequency of each word across the dataset and sort them in descending order.

How about title per word?

To train a topic model using LDA() from the topicmodel package, we need to create sparse matrix from our tidy dataframe of tokens using cast_sparse(category, word, n)

This means there are 1967 titles (i.e documents) and different tokens (i.e. terms or words) in our dataset for modeling.

A topic model like this one models:

• each document as a mixture of topics
• each topic as a mixture of words

The most important parameter when training a topic modeling is K, the number of topics. This is like k in k-means in that it is a hyperparamter of the model and we must choose this value ahead of time.

To dig deeper into our topic model, we can use the tidy() function to convert the results into a dataframe that we can work with. This gives us two types of outputs:

• Beta matrix: shows the probability of each word belonging to each topic (topic-word distribution)

• Gamma matrix: shows how much each topic contributes to each document (document-topic distribution)

We’ll start by looking at the beta matrix first.

Since the output is a tidy dataframe, we can easily manipulate it — including visualizing the top words with the highest probabilities for each topic.